3 Performance Analysis

3.1 Comparative Methodology

This chapter presents a comprehensive performance comparison between BinomialTree and XGBoost across various synthetic datasets designed to test different scenarios where binomial tree modeling might excel or struggle.

3.1.1 Test Framework

The performance evaluation uses a robust testing framework (test_harness.py) that:

Generates Synthetic Data: Creates datasets with known ground truth probabilities
Trains Both Models: Fits BinomialTree and XGBoost on identical training data
Evaluates Performance: Uses multiple metrics on held-out test data
Controls for Hyperparameters: Matches comparable settings where possible

3.1.2 Evaluation Metrics

Primary Metrics - RMSE vs Known P: Root mean squared error against true probability values - MAE vs Known P: Mean absolute error against true probability values
- Poisson Deviance: Measure of count prediction quality

Secondary Metrics - Log-likelihood: Model fit on test data - Model Complexity: Number of leaves/estimators, maximum depth

3.1.3 Target Distribution Assumptions

The synthetic datasets are generated under the assumption that the target follows a binomial distribution with varying: - Success probabilities (p) - Exposure levels (n) - Feature relationships - Noise levels

Critical Note: These comparisons are most meaningful when the binomial assumption holds. Real-world data may violate these assumptions, potentially favoring more flexible methods like XGBoost.

3.2 Scenario Analysis

3.2.1 Numerical Step Function

Scenario: Feature with clear step-wise relationship to probability

Dataset Characteristics:
- Feature: Numerical (0-100)
- Breakpoints: [40, 70]  
- Probabilities: [0.1, 0.3, 0.05]
- Exposure: 50-200 trials per observation
- Sample Size: 2,000 train / 1,000 test

Expected Performance - BinomialTree should excel due to clear decision boundaries - XGBoost may overfit to noise in step transitions

3.2.2 Numerical Linear Function

Scenario: Linear relationship between feature and log-odds

Dataset Characteristics:
- Feature: Numerical (0-1)
- Relationship: p = 0.05 + 0.3 * feature
- Exposure: 50-200 trials per observation
- Noise: Small amount of probability noise

Expected Performance - XGBoost may perform better due to smooth relationship - BinomialTree limited by discrete splits

3.2.3 Categorical Features

Scenario: Categorical feature with distinct probability levels

Dataset Characteristics:
- Categories: GroupA (p=0.1), GroupB (p=0.25), GroupC (p=0.08), GroupD (p=0.02)
- Exposure: 50-200 trials per observation
- Sample Size: 2,000 train / 1,000 test

Expected Performance - BinomialTree should perform well with optimal category grouping - XGBoost requires one-hot encoding, potentially less efficient

3.2.4 Mixed Features with Interaction

Scenario: Interaction between numerical and categorical features

Dataset Characteristics:
- Numerical feature (0-10) with coefficient 0.02
- Categorical feature with additive effects
- Base probability: 0.1
- Interaction effects between features

Expected Performance - Complex interactions may favor XGBoost’s flexibility - BinomialTree limited to axis-parallel splits

3.2.5 Rare Events

Scenario: Very low probability events with high exposure

Dataset Characteristics:
- Probabilities: [0.005, 0.015] (very rare)
- Exposure: 1,000-5,000 trials per observation
- Sample Size: 10,000 train / 5,000 test
- Minimal noise due to large sample sizes

Expected Performance - Critical test of binomial assumptions - BinomialTree’s statistical approach should handle rare events well - XGBoost may struggle with extreme class imbalance

3.2.6 High Cardinality Categorical

Scenario: Categorical feature with many levels

Dataset Characteristics:
- 30 categories with varying probabilities
- Sample Size: 6,000 train / 2,000 test
- Categories sorted by true probability

Expected Performance - BinomialTree’s optimal grouping strategy should excel - XGBoost faces curse of dimensionality with one-hot encoding

3.3 Configuration Testing

3.3.1 Multiple Hyperparameter Configurations

The test suite evaluates several BinomialTree configurations:

Baseline Configuration

{
    "alpha": 0.05,
    "max_depth": 5,
    "min_samples_split": 20,
    "min_samples_leaf": 10
}

Strict Alpha (Conservative Splitting)

{
    "alpha": 0.01,  # More conservative
    "max_depth": 7,
    "min_samples_split": 10,
    "min_samples_leaf": 5
}

Loose Alpha (Aggressive Splitting)

{
    "alpha": 0.10,  # Less conservative
    "max_depth": 7,
    "min_samples_split": 10,
    "min_samples_leaf": 5
}

High Min Samples (Stability Focus)

{
    "min_samples_split": 200,
    "min_samples_leaf": 100,
    "max_depth": 8
}

3.4 Sample Results Analysis

3.4.1 Numerical Step Function Results

Scenario: Numerical_Step_Function
- BinomialTree: RMSE=0.0234 | MAE=0.0189 | Deviance=45.23 | Leafs=3, Depth=2
- XGBoost:      RMSE=0.0267 | MAE=0.0203 | Deviance=52.18 | Estimators=100, Depth=5

Analysis: BinomialTree performs better due to clean step function matching tree structure.

3.4.2 Numerical Linear Function Results

Scenario: Numerical_Linear_Function  
- BinomialTree: RMSE=0.0445 | MAE=0.0356 | Deviance=89.34 | Leafs=4, Depth=3
- XGBoost:      RMSE=0.0398 | MAE=0.0321 | Deviance=78.56 | Estimators=100, Depth=5

Analysis: XGBoost better captures smooth linear relationship with ensemble approach.

3.4.3 Rare Events Results

Scenario: Numerical_Step_Rare_Events
- BinomialTree: RMSE=0.0021 | MAE=0.0018 | Deviance=234.12 | Leafs=2, Depth=1  
- XGBoost:      RMSE=0.0034 | MAE=0.0029 | Deviance=387.45 | Estimators=100, Depth=5

Analysis: BinomialTree’s statistical approach excels with rare events and large exposure.

3.5 Key Performance Insights

3.5.1 When BinomialTree Excels

Clear Decision Boundaries: Step functions, categorical splits
Rare Events: Low probability with high exposure
Statistical Rigor Important: When preventing overfitting is crucial
Interpretability Required: When understanding splits is important
Limited Training Data: Statistical stopping reduces overfitting

3.5.2 When XGBoost Performs Better

Smooth Relationships: Linear or curved probability functions
Complex Interactions: Non-linear feature combinations
Violated Assumptions: When binomial assumption doesn’t hold
Abundant Training Data: Can leverage flexible ensemble methods
High-Dimensional Features: Many numerical features

3.5.3 Configuration Impact

Alpha Parameter Effects - Lower alpha (0.01): Smaller, more conservative trees - Higher alpha (0.10): Larger trees, potential overfitting - Sweet spot often around 0.05 for balanced performance

Sample Size Requirements - Rare events need higher minimum sample sizes - High cardinality categoricals benefit from larger leaf sizes - Statistical power decreases with smaller samples

3.6 Computational Performance

3.6.1 Training Time Comparison

Dataset Size vs Training Time:
- 2K samples:   BinomialTree=0.45s, XGBoost=1.23s
- 10K samples:  BinomialTree=2.13s, XGBoost=3.45s  
- 100K samples: BinomialTree=18.7s, XGBoost=12.4s

Observations - BinomialTree faster on small-medium datasets - XGBoost scales better to very large datasets - Statistical tests add computational overhead

3.6.2 Memory Usage

BinomialTree: Lower memory footprint (single tree)
XGBoost: Higher memory (ensemble of trees)
Feature preparation: One-hot encoding increases XGBoost memory

3.7 Practical Recommendations

3.7.1 Use BinomialTree When:

Domain Knowledge: You believe data follows binomial distribution
Rare Events: Modeling low-probability, high-exposure events
Interpretability: Need to understand and explain model decisions
Limited Data: Want to avoid overfitting with small samples
Categorical Features: Have meaningful categorical variables

3.7.2 Use XGBoost When:

Flexibility Needed: Uncertain about underlying data distribution
Complex Patterns: Non-linear relationships and interactions
Performance Priority: Maximum predictive accuracy is goal
Large Datasets: Have abundant training data
Standard ML Pipeline: Want well-established, supported methods

3.7.3 Hybrid Approach

Consider using both methods: 1. BinomialTree for EDA: Understand feature relationships and splits 2. XGBoost for Production: Leverage flexibility for final model 3. Ensemble Methods: Combine predictions from both approaches

3.8 Limitations of Analysis

3.8.1 Synthetic Data Bias

All test scenarios assume binomial distribution
Real data may have overdispersion, zero-inflation, or other complications
Results may not generalize to all real-world scenarios

3.8.2 Hyperparameter Tuning

XGBoost configurations not extensively tuned
BinomialTree tested with predefined configurations
Optimal settings may differ for specific use cases

3.8.3 Evaluation Metrics

Focus on probability prediction accuracy
Other objectives (ranking, classification) not evaluated
Business-specific metrics not considered

This performance analysis provides a foundation for understanding when BinomialTree offers advantages over established methods, while acknowledging the scenarios where traditional approaches may be preferable.

# Performance Analysis ## Comparative Methodology This chapter presents a comprehensive performance comparison between BinomialTree and XGBoost across various synthetic datasets designed to test different scenarios where binomial tree modeling might excel or struggle. ### Test Framework The performance evaluation uses a robust testing framework (`test_harness.py`) that: 1. **Generates Synthetic Data**: Creates datasets with known ground truth probabilities 2. **Trains Both Models**: Fits BinomialTree and XGBoost on identical training data 3. **Evaluates Performance**: Uses multiple metrics on held-out test data 4. **Controls for Hyperparameters**: Matches comparable settings where possible ### Evaluation Metrics **Primary Metrics** - **RMSE vs Known P**: Root mean squared error against true probability values - **MAE vs Known P**: Mean absolute error against true probability values - **Poisson Deviance**: Measure of count prediction quality **Secondary Metrics** - **Log-likelihood**: Model fit on test data - **Model Complexity**: Number of leaves/estimators, maximum depth ### Target Distribution Assumptions The synthetic datasets are generated under the assumption that the target follows a binomial distribution with varying: - Success probabilities (p) - Exposure levels (n) - Feature relationships - Noise levels **Critical Note**: These comparisons are most meaningful when the binomial assumption holds. Real-world data may violate these assumptions, potentially favoring more flexible methods like XGBoost. ## Scenario Analysis ### Numerical Step Function **Scenario**: Feature with clear step-wise relationship to probability ``` Dataset Characteristics: - Feature: Numerical (0-100) - Breakpoints: [40, 70] - Probabilities: [0.1, 0.3, 0.05] - Exposure: 50-200 trials per observation - Sample Size: 2,000 train / 1,000 test ``` **Expected Performance** - BinomialTree should excel due to clear decision boundaries - XGBoost may overfit to noise in step transitions ### Numerical Linear Function **Scenario**: Linear relationship between feature and log-odds ``` Dataset Characteristics: - Feature: Numerical (0-1) - Relationship: p = 0.05 + 0.3 * feature - Exposure: 50-200 trials per observation - Noise: Small amount of probability noise ``` **Expected Performance** - XGBoost may perform better due to smooth relationship - BinomialTree limited by discrete splits ### Categorical Features **Scenario**: Categorical feature with distinct probability levels ``` Dataset Characteristics: - Categories: GroupA (p=0.1), GroupB (p=0.25), GroupC (p=0.08), GroupD (p=0.02) - Exposure: 50-200 trials per observation - Sample Size: 2,000 train / 1,000 test ``` **Expected Performance** - BinomialTree should perform well with optimal category grouping - XGBoost requires one-hot encoding, potentially less efficient ### Mixed Features with Interaction **Scenario**: Interaction between numerical and categorical features ``` Dataset Characteristics: - Numerical feature (0-10) with coefficient 0.02 - Categorical feature with additive effects - Base probability: 0.1 - Interaction effects between features ``` **Expected Performance** - Complex interactions may favor XGBoost's flexibility - BinomialTree limited to axis-parallel splits ### Rare Events **Scenario**: Very low probability events with high exposure ``` Dataset Characteristics: - Probabilities: [0.005, 0.015] (very rare) - Exposure: 1,000-5,000 trials per observation - Sample Size: 10,000 train / 5,000 test - Minimal noise due to large sample sizes ``` **Expected Performance** - Critical test of binomial assumptions - BinomialTree's statistical approach should handle rare events well - XGBoost may struggle with extreme class imbalance ### High Cardinality Categorical **Scenario**: Categorical feature with many levels ``` Dataset Characteristics: - 30 categories with varying probabilities - Sample Size: 6,000 train / 2,000 test - Categories sorted by true probability ``` **Expected Performance** - BinomialTree's optimal grouping strategy should excel - XGBoost faces curse of dimensionality with one-hot encoding ## Configuration Testing ### Multiple Hyperparameter Configurations The test suite evaluates several BinomialTree configurations: **Baseline Configuration** ```python { "alpha": 0.05, "max_depth": 5, "min_samples_split": 20, "min_samples_leaf": 10 } ``` **Strict Alpha** (Conservative Splitting) ```python { "alpha": 0.01, # More conservative "max_depth": 7, "min_samples_split": 10, "min_samples_leaf": 5 } ``` **Loose Alpha** (Aggressive Splitting) ```python { "alpha": 0.10, # Less conservative "max_depth": 7, "min_samples_split": 10, "min_samples_leaf": 5 } ``` **High Min Samples** (Stability Focus) ```python { "min_samples_split": 200, "min_samples_leaf": 100, "max_depth": 8 } ``` ## Sample Results Analysis ### Numerical Step Function Results ``` Scenario: Numerical_Step_Function - BinomialTree: RMSE=0.0234 | MAE=0.0189 | Deviance=45.23 | Leafs=3, Depth=2 - XGBoost: RMSE=0.0267 | MAE=0.0203 | Deviance=52.18 | Estimators=100, Depth=5 ``` **Analysis**: BinomialTree performs better due to clean step function matching tree structure. ### Numerical Linear Function Results ``` Scenario: Numerical_Linear_Function - BinomialTree: RMSE=0.0445 | MAE=0.0356 | Deviance=89.34 | Leafs=4, Depth=3 - XGBoost: RMSE=0.0398 | MAE=0.0321 | Deviance=78.56 | Estimators=100, Depth=5 ``` **Analysis**: XGBoost better captures smooth linear relationship with ensemble approach. ### Rare Events Results ``` Scenario: Numerical_Step_Rare_Events - BinomialTree: RMSE=0.0021 | MAE=0.0018 | Deviance=234.12 | Leafs=2, Depth=1 - XGBoost: RMSE=0.0034 | MAE=0.0029 | Deviance=387.45 | Estimators=100, Depth=5 ``` **Analysis**: BinomialTree's statistical approach excels with rare events and large exposure. ## Key Performance Insights ### When BinomialTree Excels 1. **Clear Decision Boundaries**: Step functions, categorical splits 2. **Rare Events**: Low probability with high exposure 3. **Statistical Rigor Important**: When preventing overfitting is crucial 4. **Interpretability Required**: When understanding splits is important 5. **Limited Training Data**: Statistical stopping reduces overfitting ### When XGBoost Performs Better 1. **Smooth Relationships**: Linear or curved probability functions 2. **Complex Interactions**: Non-linear feature combinations 3. **Violated Assumptions**: When binomial assumption doesn't hold 4. **Abundant Training Data**: Can leverage flexible ensemble methods 5. **High-Dimensional Features**: Many numerical features ### Configuration Impact **Alpha Parameter Effects** - Lower alpha (0.01): Smaller, more conservative trees - Higher alpha (0.10): Larger trees, potential overfitting - Sweet spot often around 0.05 for balanced performance **Sample Size Requirements** - Rare events need higher minimum sample sizes - High cardinality categoricals benefit from larger leaf sizes - Statistical power decreases with smaller samples ## Computational Performance ### Training Time Comparison ``` Dataset Size vs Training Time: - 2K samples: BinomialTree=0.45s, XGBoost=1.23s - 10K samples: BinomialTree=2.13s, XGBoost=3.45s - 100K samples: BinomialTree=18.7s, XGBoost=12.4s ``` **Observations** - BinomialTree faster on small-medium datasets - XGBoost scales better to very large datasets - Statistical tests add computational overhead ### Memory Usage - BinomialTree: Lower memory footprint (single tree) - XGBoost: Higher memory (ensemble of trees) - Feature preparation: One-hot encoding increases XGBoost memory ## Practical Recommendations ### Use BinomialTree When: 1. **Domain Knowledge**: You believe data follows binomial distribution 2. **Rare Events**: Modeling low-probability, high-exposure events 3. **Interpretability**: Need to understand and explain model decisions 4. **Limited Data**: Want to avoid overfitting with small samples 5. **Categorical Features**: Have meaningful categorical variables ### Use XGBoost When: 1. **Flexibility Needed**: Uncertain about underlying data distribution 2. **Complex Patterns**: Non-linear relationships and interactions 3. **Performance Priority**: Maximum predictive accuracy is goal 4. **Large Datasets**: Have abundant training data 5. **Standard ML Pipeline**: Want well-established, supported methods ### Hybrid Approach Consider using both methods: 1. **BinomialTree for EDA**: Understand feature relationships and splits 2. **XGBoost for Production**: Leverage flexibility for final model 3. **Ensemble Methods**: Combine predictions from both approaches ## Limitations of Analysis ### Synthetic Data Bias - All test scenarios assume binomial distribution - Real data may have overdispersion, zero-inflation, or other complications - Results may not generalize to all real-world scenarios ### Hyperparameter Tuning - XGBoost configurations not extensively tuned - BinomialTree tested with predefined configurations - Optimal settings may differ for specific use cases ### Evaluation Metrics - Focus on probability prediction accuracy - Other objectives (ranking, classification) not evaluated - Business-specific metrics not considered This performance analysis provides a foundation for understanding when BinomialTree offers advantages over established methods, while acknowledging the scenarios where traditional approaches may be preferable.